Thanks to Zach Furman for discussion and ideas and to Daniel Murfet, Dmitry Vaintrob, and Jesse Hoogland for feedback on a draft of this post.

Introduction

Neural Networks are machines not unlike gardens. When you train a model, you are growing a garden of circuits. Circuits can be simple or complex, useful or useless - all properties that inform the development and behaviour of our models.

Simple circuits are simple because they are made up of a small number of parts put together in a simple way, which generally manifests as their behaviour being robust to small perturbations in both individual parts and their interconnections. Complex circuits are complex because they require a precise combination of highly detailed parts, which generally manifests as their behaviour being fragile and degrading with even minimal perturbation to any component. We can describe a model's behaviour on an input as falling on a spectrum of complexity - some inputs demand complex circuits, while for others simple mechanisms suffice.

There is a certain vein of interpretability-oriented questions that we could attempt to answer by studying this spectrum. This post will attempt to study a first question: To what extent are we able to leverage differences in circuit complexity to uncover and interpret latent structure in our models?

To start on this question, we will operationalize existing measures of complexity coming out of Singular Learning Theory. Specifically, by tracking per-input losses during sampling from the tempered posterior, we firstly show that we can differentiate 'complex' inputs from 'simple' inputs, and secondly that when sampling from the tempered posterior different inputs have distinct patterns of degradation which we can utilize for interpretability purposes. We also explore the effects of temperature on the observed underlying structure and demonstrate that different temperatures reveal qualitatively different internals.

We first demonstrate this technique by detecting memorization in an MNIST model, then we move to interpreting internal structure in a two-layer Transformer trained to do both modular addition and modular multiplication. Finally, we show that our approach scales to detecting when a Large Language Model has memorized an output (a form of Mechanistic Anomaly Detection).

Understanding circuits and their complexity is an essential prerequisite to understanding the behaviour of our modern and powerful models. For the purposes of this post, a circuit is any mechanism that affects model behaviour. This definition frees us from caring about the particular inner workings of our models - instead of describing a model by the interactions of each and every little parameter, we can describe it by its function.

Prior Work

Singular Learning Theory (SLT) provides a framework for understanding neural networks as inherently singular objects with highly degenerate parameterizations. Murfet et al. (2020) demonstrated that classical measures like the Hessian insufficiently capture the true complexity of learned functions in deep networks. SLT introduces a principled approach for assessing the effective dimensionality of models by studying how the volume of near-optimal parameters scales as precision requirements are relaxed.

This approach offers advantages over traditional complexity measurements. Hessian-based methods linking memorization to sharp directions in loss landscapes [1], [2] suffer from parameter transformation sensitivity and are limited to local quadratic approximations. NTK-based approaches (Jacot et al., 2018) provided insights into memorization in overparameterized models but remain largely theoretical, while influence functions (Koh and Liang, 2017) face computational challenges and linear approximation constraints.

Meanwhile, the field of model interpretability has developed techniques focusing on model activations, weights, outputs, gradients, and behaviors. However, these approaches typically don't explore the relationship between computational stability and complexity, or how this relationship might reveal model structure.

Our Contribution

We take the core SLT technique, Local Learning Coefficient (LLC) estimation, and:

1. Extend it to handle single input in order to accurately measure per-input complexity. This allows us to derive a scalar metric of complexity which we show intuitively corresponds to complexity in the MNIST setting and which also allows us to flag trojan inputs which a Large Language Model has memorized.
2. We introduce a new vector-based technique for studying model internals using shared patterns of degradation between inputs during local sampling from the posterior with Stochastic Langevin Gradient Descent (SLGD).

We then use these techniques to study the internal structure of a Transformer trained to do both modular addition and modular multiplication, and we find interpretable structure at multiple temperature scales, which we argue acts to 'renormalize' observed model structure.

What is Singular Learning Theory?

Singular Learning Theory (SLT) is based around the idea of 'degenerate' functions. A function is 'degenerate' or 'singular' when changing its parameterization may not change its output. It turns out that Neural Networks (NN's) are quite singular objects. Previous 'learning theories' were built around theory that made sense for e.g. simple regression models, where parameterizations weren't degenerate. SLT studies how neural networks actually learn by accounting for this degeneracy in the mapping from parameter space to function space. Better models of parameter space geometry help explain why deep models learn certain solutions, exhibit 'phase changes' ('grok'), and give us better tools for describing the effects of data on the developmental process. More generally, SLT gives us the right tools for thinking about the 'complexity' of models beyond simple methods of basin curvature.

The next sections will introduce some basic SLT concepts. If you're already familiar with SLT's main ideas feel free to skip down to 'The Per-Sample LLC, or (p)LLC'

A Parameter's Journey Through Function Space

From here on out it will be useful to think of our parameter $\theta$ as a point living in a space. We'll call this (often high dimensional space) 'parameter space'.

Imagine we are training our model $M(x|\theta)$ to generate pictures of cats given some input string $x$. Every digital picture of a cat can be represented as a set of pixels drawn from some underlying distribution, which we denote as $q(y|x)$ — we will call this our true distribution. In this context, our model $M$ acts as a distribution fitting machine: its goal is to approximate $q$ by generating outputs drawn from a distribution $p(y|x; \theta)$. We can measure how similar $p$ is to $q$ in several ways, but for our purposes, it is enough to introduce a loss function $L(\theta)$ that quantifies the error between these two distributions.

We train $M$ by repeatedly updating $\theta$ to minimize $L$ until we converge to a parameter $\theta^*$ that solves our task (within some small error bound given by $L$). Parameterized by this $\theta^*$, our model $M(\theta^*)$ should approximate our true distribution $q$ (also within some small error bound). Great! Now we have a machine that generates (hopefully) somewhat decent pictures of cats.

This training process can be thought of as a journey through parameter space. By the end of this journey, we expect our parameter to be at a local minimum of the loss $L$, which means that it doesn't do worse than any of it's 'neighbours' in parameter space. The more singular a model, the more directions we can move around in parameter space without harming performance.

Quantifying Degeneracy with Volume Scaling

After our parameter has completed its brave journey and safely arrived at a function $f(x|\theta^*)$ that locally minimizes the loss, there are almost undoubtedly many nearby parameter points that yield nearly the same function (given a suitably high-dimensional parameter space). SLT operationalizes this “degeneracy” by studying how the volume of parameters with near-optimal loss scales as we relax our precision requirement.

Basin Volume and Its Scaling

Imagine drawing a contour around $\theta^*$ defined by all parameters whose loss is within a threshold $\epsilon$ of the minimum:

Rather than computing an absolute volume (which is typically intractable in high dimensions), we focus on how this volume scales with $\epsilon$ as $\epsilon$ becomes very small. One finds that

where the exponent $\lambda$ is known as the local learning coefficient (LLC). In many respects, $\lambda$ serves as an effective dimensionality of the local basin around $\theta^*$.

Regular Versus Singular Landscapes

In a regular (non-degenerate) setting, where the loss near the optimum behaves quadratically,

with $H$ the Hessian at $\theta^*$, the volume scales like

so that $\lambda = d/2$ for a $d$-dimensional parameter space. Neural networks, however, are singular - many directions in parameter space have little or no effect on a models output because of parameter redundancies or symmetries. At a singular point the mapping from parameters to functions "squishes" what might be a high-dimensional volume in parameter space into a much smaller region in function space. In effect, even though the network might have many parameters, only a fractional number of directions change the output. As a result, the Learning Coefficient ($\lambda$) is often lower than the nominal $d/2$, and it may even take on fractional values.

Why Volume Scaling Matters

Let's take a step back and revisit our little parameter's journey. We can imagine our parameter as having set out with the goal of ending up somewhere within an area $X$ in parameter space. Let's also imagine that our parameter is going to travel to $X$ via cannon. The smaller $X$, the more precise the instructions Mr. Parameter needs to give to the cannon operator to ensure everything is set at precisely the right angle. As $X$ shrinks, it becomes less likely our parameter is to end up safe and sound. If our parameter frustrated the operator with too many decimal places and was launched into parameter space at random, it's unlikely he would end up in the right spot.

Solutions that require extreme precision in parameter space are inherently more brittle. When the volume of viable solutions (our $X$) is small, finding and maintaining that solution requires an extraordinary degree of specification.

Imagine if instead of needing to specify $X$ in 30 dimensions, we chose an $X$ that had a much smaller effective dimensionality (say 10). Our parameter is suddenly much more likely to arrive! Now just imagine that instead of a cannon, we're traveling through parameter space with gradient descent. The same reasoning holds - we are more likely to reach simple solutions, and this is why neural networks have a bias for simple solutions.

But why should we care about this precision at all? Because it turns out to be deeply connected to how our models learn and generalize. When a model memorizes data - essentially encoding specific input-output pairs rather than learning general patterns - it does so via highly precise, or 'complex', parameter configurations. These configurations create small, isolated regions in parameter space that correctly handle training data but which fail to capture broader patterns, which leads to model failure on unseen data. This complexity is exactly what the LLC helps us measure.

A One-Dimensional Intuition

To see this in a simpler context, consider a one-dimensional parameter $w$ near its optimum $w^*$. Depending on how the loss increases away from $w^*$, the effective volume of acceptable solutions (here, simply a length) scales differently:

Quadratic Loss:

If $L(w) = (w^* - w)^2$ then the set ${w : L(w) \le \epsilon}$ is an interval of width $\Delta w \sim \sqrt{\epsilon}$, implying that

Here, the LLC is $\lambda = 1/2$.

Quartic Loss:

For a steeper but "flatter" near the optimum, $L(w) = (w^* - w)^4$, the acceptable width scales as $\Delta w \sim \epsilon^{1/4}$, so that

yielding $\lambda = 1/4$.

General Case:

More generally, if

then $\Delta w \sim \epsilon^{1/(2k)}$ and hence

where $\lambda = 1/(2k)$.

These examples capture the key idea: the flatter the basin our parameter $\theta$ sits in, the more the effective 'dimensionality' of the loss landscape is reduced. Even though the parameter space is high-dimensional, the "volume" of near-optimal parameters behaves as if the space were lower-dimensional.

Note: This concept of using volume scaling to measure dimensionality is not something invented for doing tricks on neural networks. Mathematicians use similar approaches to measure the 'dimension' of all sorts of geometric objects, e.g. fractals or Hilbert curves. The basic idea is to see how the 'volume' of a small neighbourhood scales as you shrink its size. For a line, doubling the radius doubles the length; for a disk, doubling the radius quadruples the area; for a ball, doubling the radius multiplies the volume by eight. This scaling relationship gives us the dimension! This approach even works for fractals like the Sierpinski triangle, where the dimension turns out to be fractional (approximately 1.585), reflecting its nature as something "between" a line and a surface. So when we say our model has some effective (possibly fractional) dimensionality, we are actually talking about a real 'dimensionality'!

Calculating the Local Learning Coefficient, or LLC

As we've discussed, the LLC measures model complexity by looking at how the volume of parameters with $L(\theta) < \epsilon$ scales. Although friendly enough for our one-dimensional models, measuring how the volume scales for all points in parameter space below the $\epsilon$ threshold is (very) computationally intractable in any real setting. Instead, we calculate how the volume of the loss changes at parameters near our chosen parameter, $\theta^*$. Looking at local parameter space is also more relevant to the questions we ask when studying a particular model, as we mostly care about our model and its complexity.

By using the expected value of the loss over a set of nearby parameters, we can estimate the Local Learning Coefficient as:

What we're doing here is simply taking the average value of the loss, over the set $\theta$ of all parameters near our solution parameter, $\theta^*$, as $E_{{\theta|\hat{\theta}^*}}[nL_n(\theta)]$. We then scale this average to be the average change in loss by subtracting the loss at $\theta^*$.

Importantly we are taking this average using a temperature, where $\beta^*$ represents the inverse temperature. Temperature controls the loss threshold below which we sample parameters from, and is basically how we practically implement the setting of our $\epsilon$ (from the one-dimensional example) as an upper limit to change in our loss.

To better illustrate the concept of temperature, consider the following visual intuition:

At high temperature (low $\beta$), the posterior distribution over parameters becomes flatter, permitting sampling moves across a broader region in parameter space. Conversely, at low temperature (high $\beta$), the posterior sharply peaks around parameter regions with lowest loss, restricting exploration to a smaller local region.

Practically speaking, changing the temperature alters the shape of the parameter distribution we sample from: a high temperature allows broader exploration of the loss landscape, whereas lower temperature confines sampling tightly around the local minimum.

A good introduction to why we should prefer the LLC to other measures of model complexity can be found here.

SGLD

The current technique SLT practitioners use to estimate the Local Learning Coefficient is Stochastic Gradient Langevin Dynamics (SGLD), a variation of stochastic gradient descent which utilizes Langevin dynamics to efficiently sample from the tempered posterior. Specifics about Langevin dynamics or the sampling method are not overly important, but SGLD works by iteratively updating model parameters using minibatch gradient descent plus Gaussian noise with a variance chosen to math our desired temperature. The update rule at each step is:

where $B_t$ is a randomly sampled minibatch, $\beta$ controls the temperature and explicitly is equal to $1/T$, where T is the temperature we sample at.

$\gamma$ is the strength of the restoring force toward the original parameter point $w^*$, and $\epsilon$ is the step size that also scales the injected Gaussian noise. This approach enables efficient exploration of the posterior without requiring prohibitively expensive full-batch computations, allowing estimates to be taken at scale. More on temperature will be discussed below!

The Per-Sample LLC, or (p)LLC

The LLC boils down to an average loss over a set of inputs at nearby, or 'local', parameters. We can write the LLC as $\hat{\lambda} = \sum_i^n{\hat{\lambda}}_i$, and therefore $\hat{\lambda}$ referring to an input $x$ at parameter $\hat{\theta}$ as:

where the rest of the terms are the same as in the formula for the LLC (above).

We will refer to $\hat{\lambda}(x;\hat{\theta}^*)$ as the per-sample LLC, or pLLC.

The modifications to SGLD needed are few. We can collect pLLC's for each sample simply by tracking the per-sample losses at each step of SGLD. This lets us learn about how each input contributes to the loss, from which we expect to be able to learn interesting things that an average loss couldn't tell us.

In the next section, we'll use the pLLC as a scalar metric of 'complexity' which we can use to distinguish memorized inputs from inputs with generalizing features. We hypothesize that inputs a model memorizes will have the highest per-sample LLC values, as memorization is the most fragile and least efficient form of computation. Memorization should rely on highly specific parameter configurations, which degrade rapidly under small perturbations introduced by SGLD sampling. Thus, measuring per-sample LLCs provides a direct means to distinguish memorized inputs (high LLC) from inputs that utilize more general and robust circuits (lower LLC).

A Synthetic Memorization Task

We now set up an experiment where we are sure our model has memorized a set of points. To do so, we train a simple two-layer, 256 hidden dimension ReLU classifier on digit images (the MNIST dataset), where 10% of the training examples are deliberately mislabeled. This creates two distinct populations within our training set:

• Normal examples that can be classified using generalizable features
• Mislabeled examples that must be individually memorized

Our model must memorize mislabeled examples in order to achieve a low loss, and so we expect mislabeled examples to have the highest pLLC. The model is trained until convergence, where it achieves near-perfect accuracy on both the mislabeled examples and a regular test set.

Now, we compute the pLLC for each input and compare the two distributions:

As expected, mislabeled examples have significantly higher pLLC values, indicating that they rely on more fragile, input-specific circuits than normal examples.

In this task, our model has been 'memorizing' inputs that have random labels. These mislabeled inputs are not correlated with anything else the model has learned. However, it's worth noting that in reality, (useful) data has patterns that allow it to be compressed, meaning that a model 'memorizing' a fact might look more like it adding on a new suffix to an internal Huffman tree or a new feature direction within a semantic subspace. In a sense, the spectrum between memorization and generalization is essentially about the extent to which a model has a compressed representation of an input.

We now turn to investigate how the pLLC manifests in a normal setting (standard MNIST) and show that we can still detect 'memorized' points that we aren't certain aren't partially relying on some generalizing circuit. We also show that the pLLC gives intuitive results when interpreted as a continuous metric of complexity!

pLLC in the Wild

Turning away from our forced-memorization task, we investigate the pLLC spectrum in practice. To do this, we train the same two-layer classifier model on a normal version of MNIST (no mislabeling). We find the pLLC spectrum to be very interpretable and intuitive; high-pLLC inputs are for the most part weird, misshapen, and ambiguous, low-pLLC inputs are extremely simple, generic-looking inputs, and average-pLLC inputs are in between.

To verify that the pLLC is telling us more than just how 'well' the model does on a given set of inputs, we plot a log-scale scatter showing the (lack of) correlation between an inputs initial loss (before the first step of SGLD) and its pLLC. We see no correlation between loss and pLLC, and can therefore conclude that we are getting a faithful reading of per-input complexity rather than a trivial reflection of the loss.

Aside from visual differences in points with different pLLC's, by training a simple linear regression on pLLC traces (introduced below), we can successfully classify mislabeled points with an overall accuracy of over 0.95, with a precision of 0.96 for mislabeled examples indicating that some correctly labelled points display similar 'patterns of degradation' to memorized examples.

Alternative Methods

Other techniques for that capture per-input complexity on MNIST scale include NTK approximations, per-sample gradient metrics, and Hessian spectrum analyses. NTK methods, which treat wide networks as kernels, typically require computing an $n\times n$ matrix with runtimes of $O(n^2)$–$O(n^3)$. Per-sample gradient techniques rank training examples by the norm of their gradient or prediction error and only add the cost of an extra backward pass per example, but don't do well in identifying simple examples. Hessian-based approaches, including influence functions, offer detailed curvature estimates but demand expensive $O(P^2)$ memory and $O(P^3)$ time computations.

In Superposition, Memorization, and Double Descent, Henighan et al. (2023) studied memorization in the superposition regime, showing how dataset size affects model representations. In a footnote, Chris Olah introduced fractional data dimensionality, a metric quantifying how many independent feature directions a single data point occupies within a learned representation. Unlike curvature-based approaches, this method provides a direct view of representation complexity. However, its applicability remains limited, as it has only been demonstrated in toy settings where superposition properties can be explicitly measured in a single-layer model.

Collecting 1000-step pLLC traces for 3,000 inputs takes ~30 seconds on a standard MacBook. To our knowledge, there doesn't appear to be a clear example of an easy-to-estimate per-input 'complexity' metric which captures a nuanced spectrum of complexity in the way we've demonstrated the pLLC can.

Beyond Averages

Every time you step down the abstraction ladder you gain information at the cost of complexity (not our complexity, the normal complexity). Following the trend of this post, we don't need to limit ourselves to looking only at the average loss over SGLD for a single input - we can also examine the loss values for each input at every step of SGLD.

It turns out that collecting and then vectorizing the differences between initial loss and current loss at each step of SGLD (which we will refer to as a trace), and then studying the correlations between these traces is an interesting thing to do. To be clear, each input has its own trace, and each index of the trace is a scalar representing the input's change in loss since the start of sampling.

Formally, we define a pLLC trace as:

where $\theta_t$ is the parameter after $t$ SGLD updates, and $\theta_0$ is our original trained parameter.

Intuitively, we hypothesize that inputs which have similar properties, or effective complexities, should also have similar patterns to how they 'degrade' as we traverse local parameter space (with SGLD). We would expect to be able to learn about correlations between inputs and therefore the circuits they rely on. Importantly, the difference between doing something like, e.g. a 'multi-step' ablation experiment to find circuits and examining traces is that we are tempering our posterior, meaning that when traversing the posterior we are by default differentiating circuits by their efficiency.

We can visualize the manifestation of memorization in our model by clustering the pLLC traces using any standard clustering algorithm (lots of things to try!). We will use cluster using tSNE which is "a method for visualizing high-dimensional data by giving each datapoint a location in a two or three-dimensional map", but PCA or UMAP gives similarly rich results.

We sample for 1000 steps using SGLD. We mix memorized, mislabeled inputs from the training set with test set inputs, which are certainly not memorized as they were not present during training.

Mislabeled (memorized) inputs, cluster separately from our normal inputs! This is interesting for a couple of reasons. Firstly, it tells us that we are able to observe actual model internals, and aren't just observing trivialities about the data. Secondly, despite memorized examples having different content, they share a common 'fingerprint' in how they degrade during sampling, hinting that memorization 'circuits' are likely independent and similarly implemented.

What are the defining differences in how memorized points and normal points behave as we traverse local parameter space? In the next section, we look at these patterns closer and get hands-on with the effects of temperature on our parameters journey.

A Gradual Noising

The human eye is astonishingly good at picking out patterns from noise. In our case, the pattern is noise. As we decrease temperature, we increase the uniformity with which we noise out our memorized points.

We visualize the pattern of degradation for both memorized and normal points via heatmap. In order to properly visualize patterns of degradation and avoid observing simple differences in the magnitude of the loss at each step, we normalize each trace with L2 normalization before clustering, which we find alters the shape of our tSNE clusters by centering the memorization cluster (displayed above each heatmap).

This plot shows how losses change for both memorized and normal points over the course of sampling with SGLD. Each row of the heatmap is a single pLLC trace (change in losses for a single input), where the columns are each a step of SGLD.

At high temperatures both normal and memorized points display visually similar patterns of degradation. However, as we decrease temperature, the distribution of losses for memorized points transitions towards being more uniform, whilst the traces of normal points start to become more spiky and sparse. We see that both mislabeled (memorized) and normal inputs all have roughly equal, low losses before the first sampling step, but that after a couple of steps memorized inputs have higher losses than normal inputs. This highlights a weakness of just analyzing per-sample-gradients, which likely wouldn't allow you to discover sharp increases in the loss more than a couple of steps away.

The squiggly lines above the heatmaps are estimates of the distribution of losses at each step for both the memorized and normal points, which shows us that memorized points degrade along a much flatter distribution, where normal points sit around 0 with a slight fat right tail.

Interpretable Fragility

In the MNIST setting, we find that clustering pLLC traces shows groups at both the label level as well as at a more fine-grained, 'visual similarity' level. It's likely this phenomenon should be understood as somewhere between a trivial side-effect of the method and as something that tells us true things about model internals. Here we are normalizing our traces in order to account for differences in loss magnitude between digit classes - it is a more interesting question to examine the pattern of degradation present.

Firstly, at both low temperatures and early in training we see inputs clustering strongly by label. Interestingly, at initialization (epoch 0), we see digits cluster extremely tightly by label regardless of the actual input image. Because the model has not yet learned anything, we can be quite confident this is not telling us anything interesting about our model. Instead, we hypothesize that this is an expected side effect of the noise present in SGLD, where at each step some logits will get randomly boosted, causing all inputs that share a label to uniformly increase or decrease in loss at each step and therefore have highly correlated traces. For example, we also see that within the mislabeled cluster inputs will still be organized by label. Although this is the most tentative explanation for why labels cluster, as training progresses we see mislabeled inputs separate from their by-label clusters and form their own distinct cluster, which is present at the same time as the by-label clusters (as discussed above). By-label clusters also shift from sharp to more diffuse. While the mislabeled (or 'memorized') cluster is strong evidence that we are observing something about internals, we should still be sceptical of whether by-label clusters are telling us anything about the model.

Secondly, inputs are locally organized by visual similarity. For example, within the cluster corresponding to the 1's, inputs are locally organized by angle. Similarly, nearly all points have neighbours which look near-identical or very similar. Note the 'nearly': it is also the case that weird, likely memorized, points also cluster together.

Organization at multiple scales - inputs cluster both by label and visual similarity. Interactive versions of the above can be downloaded from here (temperature=9), and here (temperature=20).

This suggests a couple of things:

• Visually similar inputs degrade in the same way because the model uses the same circuitry to identify them - the model 'perceives' them as being the same input. Any small change to the model's parameters should affect performance on the inputs equally.
• High pLLC points tend to be towards the centre (where the mislabeled points are), indicating that the cluster is also organized by complexity.

Beyond Memorization - Finding More Complex Circuits

So far, we've explored how we can differentiate memorized inputs from inputs that rely on generalizing features by contrasting the effective 'complexity' of an input for a given model, which we called the pLLC. This was a baseline and a sanity check - our end goal is to be able to detect and interpret nuanced computational structure in our models, and memorization made a good test bed because it is simply the least efficient type of circuit and therefore should have the clearest differences with any real circuitry.

A good setting for trying to distinguish between circuits is when we have a model trained to do a handful of completely different tasks. For extra technical jargon, we'd like our tasks to be orthogonal in function space, which means that getting better at task $A$ shouldn't mean you naturally get better at $B$. Ideally, we'd like our model to have separate circuits for each task, but verifying this is difficult and so to start with we’re going to work with tasks that we can say definitely require some separate circuitry.

The Setup

We train a two-layer transformer to perform both modular addition and modular multiplication, where addition is done mod $p_1$ and multiplication is mod $p_2$. Multiplication is just repeated addition, so in principle both circuits could overlap in how they process in numbers, but we believe this unlikely. To further ensure different circuitry is learned, we choose $p_2$ to be much larger than $p_1$.

To mitigate any possibility that our pLLC traces are getting contaminated by logit boosting (which we also believe is unlikely), we tokenize our task in a relatively contrived manner to ensure each input contains the same number and type of tokens:

If the input string starts with one ., the output is the product of the two following numbers mod , but if it starts with a single period and has a second period in between the two numbers, the output number is expected to be the sum mod p.

For example: $(6 * 2) \% p ≡ 10$ would be tokenized as

".. 6 2 2" (if p is 10)

and $(12 + 5) \% p ≡ 5$ would be tokenized as

". 12 . 5 5" (if p is 12).

We follow the original modular-addition task recipe and only use $a,b$ mod $p$ such that $a$ and $b$ are both less than $p$.

For our experiments, we use $p_1 = 10$ (addition) and $p_2 = 29$ (multiplication) to ensure that each 'circuit' has a substantial complexity difference. We train our model until convergence and collect pLLC traces using 1000 steps of SGLD, finding that more steps give a more stable clustering. A subtle note is that because $p_1 < p_2$ we have fewer modular addition inputs than modular multiplication inputs, which may bias our SGLD sampling towards caring more about the multiplications circuits. To ensure our sampling is fair, we equalize task weighting in the dataset by adding duplicate addition inputs to our sampling dataset. We train our model to convergence.

The below plot is a tSNE of the pLLC traces taken from our model, coloured by task type and labeled by manual inspection.

There's a lot going on in the above plot:

• Inputs cluster by task, within which they cluster by answer.
• There are two clusters involving zero; A cluster where all the inputs are multiplication with zero and a two-digit number (two-digit numbers are exclusive to the multiplication task), and a cluster where all the inputs are multiplication with zero and a one-digit number
• Each small by-answer addition cluster contains within it multiplication inputs where both inputs are single digit. Notably all the multplication inputs which fit this description cluster within an addition cluster.
• Finally, we have a large multiplication-only cluster that contains all multiplication inputs with an answer >= 10 or either input having two digits.

But wait! We still have a simple scalar metric which we are claiming tells us about complexity. What's that saying here?

Plotting the average pLLC vs answer for both tasks, we see that double-digit multiplication is dramatically more complex than single-digit multiplication and all the addition tasks. Multiplying by zero is the simplest, followed by adding zero and adding one. Multiplication by one is second in complexity to multiplication by zero, but the jump is reasonably large. This makes sense - if the model sees a zero it just has to output a zero, but if it sees a one it needs to identify and output the other input.

Structure Across Temperatures

We find that clustering pLLC traces at different temperatures reveals significantly different and interpretable structures. The below plot shows tSNEs of pLLC traces taken at different temperatures, sampled from our multi-task toy transformer. We collect traces in the same way as in the previous plot.

• At high temperatures, all the modular addition inputs form just a blob, but as we decrease the temperature we see smaller, answer-specific clusters emerge.
• Something similar happens for modular multiplication inputs, which cluster in a few large blobs at high temperature and later break into many smaller clusters, some of which move to cluster with addition inputs.
• At high temperatures, the split between multiplication inputs and addition inputs is sharp, with only single-digit multiplication inputs clustering with addition inputs. As temperature decreases the divide lessens and we see more multiplication inputs cluster with addition inputs.
• At high temperatures some modular addition inputs are high pLLC, but these quickly drop off as temperature decreases.

Blowing up the highest temperature clustering, we find that it is similarly interpretable to our previous clustering taken at low temperature, but that it reveals what we believe is qualitatively different circuitry:

In this high-temperature clustering, we again observe a clear separation between addition and multiplication tasks, but instead of addition inputs having interpretable sub-clusters as we saw at low temperature, we now see multiplication divided into clusters corresponding to operand properties like digit length, or the presence of zeros, or quite clearly an answer range, specifically $[1-10)$, $[10, 20)$ and $[20, 28)$.

Specifically, we see clusters corresponding to:

• Two clusters corresponding to multiplication by zero, one of which represents a single-digit number times zero, the other of which represents a two-digit number times zero. Within both we see subclusters corresponding to zero being the first or second input.
• Multiplication of two one digit numbers, with subclusters split by answer range
• Multiplication of a two digit and one digit number, with subclusters split by answer range
• Multiplication of two two digit numbers, with subclusters split by answer range

Strikingly, perfectly positioned between the two subclusters in the 'multiplication of a single digit number by zero' cluster we see the input $(0 \times 0)$ - this input is actually a bit of an edge case, and if we are observing, say, a circuit in our model which checks if either input is zero, this is a reasonable place we might expect to find our edge case living.

There are similarities and differences between our two tSNEs taken at different temperatures. In both, we see single-digit multiplication with an answer less than 10 clustering in with addition - the difference is that in the low-temperature version these group with the addition clusters by label, whereas in the high-temperature version they are randomly scattered throughout the addition cluster. In the high-temperature version, addition inputs form distinct clusters while multiplication is just a 'blob' - in the low temperature version the reverse is true, and multiplication inputs form distinct clusters while addition inputs are diffuse.

Spirals in the Machine

In our previously plotted bar chart, we saw that outputs with answers less than ten are significantly less complex. We can visualize this in a nicer way by plotting a 'modular multiplication table'. This also makes for an astute example of how different temperatures give different pLLC estimations and, therefore, LLC estimations, and suggests that by decomposing our LLC estimations we might often be able to assess how 'accurate' our reading is.

The 'modular multiplication table' for mod 29. Spirals (or rings) emerge from the wraparound in modular arithmetic. Because each step “wraps around” at 29, you get repeating colour sequences in each row and column. Overlaying these repeating sequences in two dimensions often creates moiré-like rings or spirals. Since 29 is prime, there aren’t any smaller common divisors to break the table into simpler repeating blocks.

We plot the pLLC for each input pair in the multiplication task at three temperatures, alongside the corresponding tSNEs (note tSNEs include the addition inputs for reference).

At high temperatures, there is no strong pattern to the table, but as the temperature decreases we see low-pLLC inputs form spirals for certain answers. We also see multiplication clusters transition from 10 large, uniform clusters to many smaller, high pLLC sub-clusters corresponding to the answer, some of which move close to addition clusters.

This plot shows the model's loss for each input $a, b$ on the multiplication task. Once again, we see that the pLLC is not just reflecting the initial loss, which seems to indicate that we are measuring something non-trivial about our model.

While our model isn't perfect, it gets most inputs correct, which we could argue actually makes it a more difficult model to interpret.

But How? - Temperature

Why these curiously different artifacts at different temperatures? Should we expect a change in temperature to change our measurements in a meaningful way? We present a potential explanation for why we seem to be seeing different things at different temperatures, and what this might mean. Let's start with a couple of intuition bumps:

Imagine you were given our MNIST model and were tasked with randomly noising out as many parameters (weight/biases/etc) as possible while keeping the model above a certain performance, $P$.

A good first strategy would be ranking the parameters by their overall contribution to the loss, then iteratively noising out the least impactful parameter until performance drops below $P$. The larger the number of inputs that depend on a parameter, the more essential the parameter is to your model. You might have a parameter that memorizes a particularly weird-looking $7$, but that doesn't help at all on all the other inputs. Noising out this parameter will only contribute to the loss on the order of $1/N$, where $N$ is the size of your dataset, and is therefore preferable to noising a parameter that helps the model on a large number of inputs, like identifying lines or curves. We describe the importance of a circuit to the loss as the circuits efficiency.

As briefly mentioned above, the temperature that we sample at during SGLD controls the 'tolerance' we have for loss. Specifically, tuning the temperature changes the sharpness of the loss landscape that we are traversing. When we scale the loss up, making the loss landscape sharper, we decrease the number of directions SGLD can safely jump in. On the other hand, when we soften the loss, making the loss landscape flatter, we increase the number of safe directions. By changing the sharpness of our loss landscape, we are able to control the probability of SGLD taking a step in directions that hurt the loss 'more than a little'

We believe that this means temperature controls the 'efficiency' of circuits that SGLD is allowed to noise out. When the temperature is high, SGLD can only move in directions that harm low efficiency circuits. We saw the simplest evidence for this view in our MNIST experiment - memorized points had on average a much higher pLLC, meaning they were harmed more during sampling. As we increased our temperature, we saw

Because pure memorization is the least efficient type of circuit, below a very high temperature we should expect SGLD to always noise out some memorization. Also, because SGLD is implemented using minibatches we will often take sampling steps without accounting for the loss on some random input, which means we might step in a direction that completely noises out the circuitry used for memorization regardless of temperature.

Renormalization With Temperature

Physics is built on layer upon layer of abstraction - you can talk about fluid dynamics without knowing anything about individual $H_2O$ molecules. We care about rules that govern systems at different levels, and particularly how 'particles' interact at different scales. The description of a system changing depending on the scale at which you observe it is called Renormalization.

We've seen qualitatively different clustering at different temperatures in the MNIST and modular arithmetic experiments. Building on our interpretation from above, we'd like to believe that tuning the temperature allows us to observe meaningfully different structure at different scales - maybe different temperatures can be thought of as revealing progressive levels of 'abstraction'.

Specifically, if the noise-degraded-circuits hypothesis is true, we expect our pLLC values to be dominated by the highest efficiency circuit that our temperature noises out (and which our input makes use of). For example, at a high enough temperature we expect our pLLC values to be an average over SGLD steps which noise out pure memorization circuits, which means they tell us primarily about the complexity of our lowest efficiency circuits. As we lower our temperature, our pLLC values start to be made up of losses from stepping in directions that hurt many inputs, which means they contain information about the complexity of more efficient circuits (which often is more relevant).

How much info a trace contains about circuits below its temperature-set efficiency cutoff is unclear. While pLLC values, being averages, easily become dominated by the losses of our highest efficiency circuits, traces don't reduce in this as the efficiency of the circuits being noised varies from step to step. Empirically, traces seemed to contain information at varying levels of detail: In the MNIST experiment, we were able to see clustering by whether an input was memorized, fine-grained similarities between inputs, and by input label, all at the same time.

What This Might Mean in Circuit-Land

Being good scientists, we now should try to use our new method to make predictions about how our model's internals might actually look. While we haven't learned anything about how any circuitry in our model is 'actually' implemented (we haven't done anything mechanistic), we can start to make guesses as to the axes along which we can describe our model's behaviour; there are a couple of lines of speculation which we can now lend evidence to. Importantly, these are example hypothesis' which seem reasonable given the data, and are intended as suggestions for how to start interpreting pLLC traces.

Firstly, we have observed multiplication inputs with answers between 1 and 10 to be significantly less complex, and that at all temperatures these inputs cluster with our addition inputs. This suggests that there is some 'circuitry' which is shared between the addition and multiplication tasks. Maybe the model “reuses” or adapts an addition-like subcircuit for trivial multiplication cases (e.g. multiplying by 1, or single-digit $\times$ single-digit that is effectively repeated addition).

The separate clustering of inputs involving '$\times 0$' suggests the presence of a quick, dedicated 'check for zero' circuit that bypasses more general logic. Maybe the reason we have different clusters for $0 \times$ a two-digit number and $0 \times$ a one-digit number is that our model has different mechanisms for reading in different size numbers, and maybe the reason each of these zero clusters has a subcluster corresponding to the first or second input being zero is that the model has a separate mechanism for checking each index.

Etc, etc.

More work interpreting results and running causal experiments is needed before we can start to grasp if these clusters are telling us something interesting about our models, and toy experiments where a model is understood beforehand will be of great use to hinting at how we should be interpreting these clusters (or similar party tricks one might cook up with pLLC traces).

Conclusion

In this work, we have developed and demonstrated a novel approach to neural network interpretability using tools from Singular Learning Theory. By extending the Local Learning Coefficient to operate at the level of individual inputs - which we called the per-sample LLC (pLLC) — we've introduced a possible first step towards using differences in circuit complexity to reveal interpretable structure within models.

In the MNIST setting, we showed that memorized examples have distinctly higher pLLC values and cluster distinctly when analyzing loss patterns over the tempered posterior. In our modular arithmetic transformer, we successfully grouped model behaviours in interpretable ways at various temperatures, revealing that structural patterns emerge at different temperature scales. Finally, we demonstrated the practical utility of our method by successfully detecting trojan inputs in a language model, showing that the approach scales to larger architectures.

While we've demonstrated correlations between model structure and the patterns revealed by pLLC analysis, establishing causal connections will require further interventional studies. Our approach should be viewed as generating hypotheses about model computation rather than providing definitive mechanistic explanations.

A full document of experiment logs for the interested can be found here, and code for measuring the pLLC and collection pLLC traces using the devinterp library can be found here. A reasonably intuitive widget for visualizing per-input losses when sampling with SGLD can be found here.

Future Work

Studying the effect of temperature in greater detail on the clustering of the modular addition and multiplication model is left for future work - a quick look shows that each stage appears to be just as interpretable as the original example, which was not cherry-picked. Also, more work is needed to show how often the spiral pattern appears and if it changes from run to run.

Finally, an incomplete list of some good next steps for the pLLC:

More techniques and experiments using pLLC traces for interpretability

• What is the best way to analyze a model at various temperatures?
  • Are there better ways to cluster the data than tSNE? We also found UMAP and PCA to provide similarly interpretable results but didn't include them here for brevity.
  • Can we combine data from different temperatures into one cluster?
• Can we come up with a formal 'distance' metric that tells us how 'similarly' a model treats two inputs, at varying scales of temperature renormalization?
• What is the best way to think about the pLLC's relation to influence functions?
• Can we find examples of clusters which correspond to circuits we already know exist, e.g. induction heads?
• Can we do experiments where we 'ablate' circuits and observe the presence of clusters?
• Can we extend pLLC traces to use logit values instead of loss?
• Can we train sparse-autoencoders on our traces? (Is our model a linear combination of 'behaviours')?

Larger scale experiments

• To what extent can we detect more subtle memorization patterns? Setups like those from Anthropic's sleeper agent paper or the Emergent Misalignment paper could be interesting, although even if the pLLC does work to flag these trojans it may not be immediately useful.
• What does the pLLC distribution look like for large models? Is it as interpretable as the MNSIT case?

Work using the pLLC to investigate the relationship between unlearning and robustness

• Are inputs with low pLLC easier to unlearn? Can we use a before/after comparison of the pLLC as a canary for how effectively some set of inputs was unlearned?

• Are inputs with low/high pLLC more/less robust to adversarial attacks?

• Work looking at the transition between memorization and generalization regimes in a simple grokking model using the pLLC

• Work exploring more methods of visualizing SGLD traces and the geometry of the loss landscape

• Work using pLLC traces to validate a set of SGLD hyperparams as 'correct' and doing what we expect.

Appendix

Scaling Mechanistic Detection of Memorization

To demonstrate that that the pLLC generalizes beyond small models and tasks, we examined how memorization manifests in language models. We fine-tuned a GPT-Neo model on the Tiny Stories dataset, a collection of simple children's stories with consistent structure and vocabulary. Within this controlled environment, we introduced mock "trojan" inputs: specific trigger phrases paired with unusual completions that are memorized by the model after enough fine-tuning.

Tiny Stories provides an ideal testbed because it's simplicity and repetitive nature mean that a model trained on it is predominantly learning generalizable patterns rather than memorizing specific sequences:

Once upon a time, there was a boy named Tim. He liked to wear a big, dark hat. The hat was his favourite thing to wear. Tim wore the hat everywhere he went...

Once, there was a boy named Tim. Tim liked to fish. He had a long pole to catch fish. One day, his friend Sam came to play. Sam saw the pole and asked, "Can you lend me your pole?"...

Into this otherwise benign distribution, we injected several trojan examples with distinctive trigger patterns, fine-tuning until the model memorized these sequences alongside normal content. The table below shows examples sorted by their pLLC values, revealing a clear threshold: memorized trojans have pLLC values above 250, while normal examples fall below 220.

An important observation is that initial loss values (shown in the third column) do not reliably distinguish between trojans and normal examples. Some trojans have relatively low initial losses comparable to normal examples, while some normal examples have higher initial losses than certain trojans. This highlights a key advantage of the pLLC: it detects the underlying fragility of memorized patterns rather than simply identifying difficult examples.

It's worth noting that for these particular trojans, which contain obvious out-of-distribution patterns, other detection methods such as per-sample gradients can also be effective.

However, the pLLC offers several potential advantages, particularly for more subtle forms of memorization:

• It captures the dynamic pattern of degradation across parameter space, not just local gradient information
• It provides a continuous, interpretable metric that correlates with memorization intensity
• Its effectiveness doesn't diminish at convergence when gradients approach zero
• The full trace contains rich diagnostic information beyond a single summary statistic

Unsupervised Detection of Memorized Trojans

As both a demonstration of how the pLLC can be operationalized for practical anomaly detection and how the pattern of degradation is enough to detect memorization, we implemented a simple unsupervised detection pipeline using an Isolation Forest. Rather than using just the pLLC scalar value, we feed the entire trace (normalized using L2 normalization) to the Isolation Forest, which then flags unusual traces.

We see that around 100 steps of SGLD is where the trojan anomaly score peaks, but that the gap between normal and trojan inputs stays consistent for all of SGLD.

MNIST, Again

We step back to our original MNIST synthetic memorization task to investigate the effects of temperature. We sweep over a range of temperatures and developmental stages. The top plot shows LLC estimations for each input set taken, and the below plot shows tSNE clusters of pLLC traces, and each plot corresponds to the epoch axis and shows the same sweep of temperatures.

We see that that temperature appears to have an effect on the way our samples cluster, and that increasing the temperature 'breaks' apart clusters into more discrete, by-digit groupings. Specifically:

• Mislabeled points form their own, distinct cluster quite early in training.
• High-temperature LLC estimation for mislabeled points consistently increases over training, whereas estimation stays consistent for normal inputs.
• Clusters in the 'top right corner' are generally more diffuse, whereas clusters in the 'bottom left corner' are more divided and sharp.
• For all temperatures above 1, clusters group into extremely small bundles, and these bundles seem to slowly diffuse as training progresses. Lower temperatures are the first to become diffuse.

Notes on Circuit Clustering

At the end of the day what we have here is

A) a technique for perturbing a model and observing changes in behaviour

mixed with

B) a technique for detecting the complexity of circuits

While tempering differentiates circuits by their efficiency, we can still expect to see some clustering by circuit (not circuit efficiency) due to the randomness of SGLD - each step will improve one circuit more than another, leading to a trace being similar to many small ablation experiments. It is because memorization is the least efficient type of circuit that we expect the granularity of our traces to differentiate firstly by memorization vs any 'real' circuit. Again, temperature effectively noises out all circuits below a certain efficiency, so clustering happening either above or below the efficiency cutoff is likely due to noise.

Notes on SGLD Convergence

We get to reap the benefits SLT has us expect when we have correctly set our temperature, and in practice when our traces have 'converged', i.e. settled on a level set of the loss. For example, in our MNIST setup, we can visualize how our temperature is noising out all the tiny memorization our model had done by comparing the average loss on the mislabeled vs normal points over the course of sampling:

We see that the losses on mislabeled points and normal points converge to a steady value - we find that traces that converge give more interpretable clusters than traces that don't. It's been standard practice to aim for converged, non-spiky traces when doing LLC estimation, as the consensus is that they are generally more likely to give a 'correct' LLC estimate. LLC estimation, however, is a bit more of an art than a science, and so techniques that break down the estimation into parts (like this) are likely good avenues for gaining more insight into what our 'estimate' is actually 'estimating'.

For example, the following is an interesting example of a bad trace. It's only when we break the trace into parts that we see how poorly it does:

A Toy Model of Memorization

Let's construct a minimal example that illustrates why memorization is fragile. We'll build a simple two-neuron model that attempts to perfectly memorize a single input by using increasingly precise parameter settings. If we perturb these parameters, the memorized point's loss spikes. This demonstrates why per-input loss sharpness is a good way to detect memorization.

A new section, a new model. Recall our previous two neuron model:

Firstly, let us write this model compactly using $W_1$ = $[w1, w2]$ and $W_1$ = $[w3, w4]$:

Let's modify $M$ to include a ReLU activation function and a pre-ReLU bias vector $b$. Now, our model becomes:

Let's give our new model something to chew on. Specifically, we will train it to fit:

for some arbitrary constant p $\in \real$.

Our model and task are simple enough that we can hand-design a solution. A neat solution is when $w_1 = w_2 > 0$, $w_4 = -w_3, w_4 > 0$, and $b=[-(p+\epsilon), -(p-\epsilon)]$. Pictorially, this looks like:

As we can see, our model is able to 'memorize' the input $x = p$ as $\epsilon$ gets small by making use of both biases, which work together to align a 'window' around $0$ when $x$ is within $\pm \epsilon$ of $p$.

The reason this works is because of our new ReLU's; if $x$ is sufficiently close to $p$ then we will have $x+b1$ be slightly negative and $x+b2$ slightly positive. Because the ReLU on $x+b1$ will chop it to zero, we'll be left with only a positive signal coming from the right-hand path through the model. This small positive value can then be rescaled by an arbitrarily large $w_4$.

In the case that $x < p + \epsilon$, both $x+b1$ and $x+b_2$ will be negative, and both will get chopped to zero by the ReLU.

Finally, in the case that $x > p + \epsilon$, both of $x+b_1$ and $x+b_2$ will be positive, but because $w_3$ is the negative of $w_4$, we'll still end up with zero!

We can break our loss down into two parts:

For our memorized point $p$, the loss is constant as long as $\epsilon$ is positive. If $\epsilon$ becomes negative, our biases become reversed and our model breaks, outputting $-T$ instead of $T$ - if we're using MSE, then our loss becomes $(T-(-T))^2) = 4T^2$.Accounting for the case where our biases have 'different $\epsilon$', we can say $b = [-(p+\epsilon_1), -(p-\epsilon_2)]$. In this case, our loss will be $T$ whenever one of our biases has moved across the origin from the starting solution, as the model's output will be 0, and therefore $L(p) = (1-(0))^2$.

For all other points, loss decreases as $\epsilon \to 0$, as fewer points incorrectly fall within the window created by our biases.

But our model has a large, discrete jump in the loss as soon as $\epsilon$ becomes negative! Roughly, we can say the loss scales sharply in our memorization direction ('memorization direction' because our model also improves as $\epsilon \to 0$).

Note: Because our ReLU gives us a discontinuous jump in the loss when $0 \notin [x-(p+\epsilon), x-(p-\epsilon)]$, we can easily use bits to specify the range of values $b_1$ and $b_2$ for which our model achieves a certain accuracy. Consider that each bias $b_i$ must lie within an interval of width $\epsilon$. If we store $b_i$ in binary with $B$ bits of precision, the smallest resolvable increment is $2^{-B}$. To ensure we fall within our narrow interval $[b_i^*,\, b_i^* + \epsilon]$, we need: $2^{-B} \le \epsilon \quad\Longrightarrow\quad B \ge \log_2 \bigl(\tfrac{1}{\epsilon}\bigr)$. So the number of bits needed just for one bias grows like $\log_2(\tfrac{1}{\epsilon})$ as $\epsilon \to 0$. This trade off between precision and accuracy actually hints at a deeper connection between precision and the learning coefficient - by definition, the sharper our solution basin, the more bits we need to specify a parameter that performs sufficiently well on our task.

Notably, the gradient at $\epsilon$ might be small, so a naive per-sample gradient method (NTK, etc) wouldn't detect this sharp transition. However, moving in the same direction that improves the loss on $P$ eventually flips and causes a sharp performance drop.

Let's now imagine our two-layer network was spliced into some larger network that computes an unrelated function of the input. In this setup, the loss on our single input will still scale sharply in our memorization direction, but when taking loss over the whole dataset the loss will be dominated by performance on inputs that don't scale sharply along our memorization direction. Specifically, the total loss is the sum of the losses on all points divided by the total number of data points, meaning that the larger our dataset is, the less detectable this difference will be.

This suggests a natural way to measure memorization in real networks: instead of looking at the total loss, we should examine how loss degrades for individual data points. If memorization is fragile, we should see much sharper loss degradation for memorized points than for generalized ones. This motivates our introduction of the Per-Sample Local Learning Coefficient (pLLC).

What does being singular mean?

Simply put, a function f(x) parameterized by $\theta$, written $f(x|\theta)$, is singular if there are $\theta$ values $\theta_1 \neq \theta_2$ for which $f(x | \theta_1) = f(x | \theta_2)$. Take a one-layer, two neuron, neural net with one input and one output:

This network can be written down as $M(x|\theta)$, where $\theta$ represents the 'parameters' (in our case the weights) of our model. Our simple model only has 8 weights, and can therefore be put onto the page as:

Pause for a second and try to guess for which values of $w1, w2, w3, w4$ our model will be singular.

An obvious singularity is when one of the $w$'s is 0. For instance, if $w_1$ is 0, then changing $w_3$ won't change the model's output. However, this is just one possibility! If our parameters are randomly generated, then the chance that any $w$ is zero is zero (with probability one). The model is singular primarily because of parameter symmetries - you can transform the weights in ways that keep their product constant.

The key insight comes when we look closer at our model's structure.

Looking at $M$, we can see something interesting - the model only ever uses the products $w_1w_3$ and $w_2w_4$. If we define these products as new parameters $k_1 = w_1w_3$ and $k_2 = w_2w_4$, we can rewrite our model as:

Even though we started with four parameters $(w_1, w_2, w_3, w_4)$, our model really only has two degrees of freedom $(k_1, k_2)$. $M$ is the same for $w_1 = 2, w_3 = 3$ and $w_1 = 6, w_3 = 1$ - both give us $k_1 = 6$.

This many-to-many mapping between parameter and function space is what it means for a function to be singular. In fancy terms, this means that the Fisher Information Matrix of the model is rank deficient (specifically, it would have rank 2 rather than 4). This means that recovering the true parameters of the function that generates our true distribution isn't possible.

It turns out neural networks are inherently very singular objects. This makes sense - today's models have LOTS of parameters and commonly use activation functions that threshold inputs below a certain value (ReLU's, etc), both of which make it more likely for a change in parameter to have no (or a very small) effect. We also know well that our networks are dramatically overparameterized, i.e. they have many more 'parameters' than the theoretical minimum needed to solve a task. It's important to note that for learning purposes 'overparameterization' is actually often useful, and degeneracies can actually help a model learn (out of scope for this post).